An evaluation of the Microsoft HoloLens for a manufacturing-guided assembly task

Many studies have confirmed the benefits of using Augmented Reality (AR) work instructions over traditional digital or paper instructions, but few have compared the effects of different AR hardware for complex assembly tasks. For this research, previously published data using Desktop Model Based Instructions (MBI), Tablet MBI, and Tablet AR instructions were compared to new assembly data collected using AR instructions on the Microsoft HoloLens Head Mounted Display (HMD). Participants completed a mock wing assembly task, and measures like completion time, error count, Net Promoter Score, and qualitative feedback were recorded. The HoloLens condition yielded faster completion times than all other conditions. HoloLens users also had lower error rates than those who used the non-AR conditions. Despite the performance benefits of the HoloLens AR instructions, users of this condition reported lower net promoter scores than users of the Tablet AR instructions. The qualitative data showed that some users thought the HoloLens device was uncomfortable and that the tracking was not always exact. Although the user feedback favored the Tablet AR condition, the HoloLens condition resulted in significantly faster assembly times. As a result, it is recommended to use the HoloLens for complex guided assembly instructions with minor changes, such as allowing the user to toggle the AR instructions on and off at will. The results of this paper can help manufacturing stakeholders better understand the benefits of different AR technology for manual assembly tasks

An evaluation of the Microsoft HoloLens for a manufacturing-guided assembly task

Many studies have confirmed the benefits of using Augmented Reality (AR) work instructions over traditional digital or paper instructions, but few have compared the effects of different AR hardware for complex assembly tasks. For this research, previously published data using Desktop Model Based Instructions (MBI), Tablet MBI, and Tablet AR instructions were compared to new assembly data collected using AR instructions on the Microsoft HoloLens Head Mounted Display (HMD). Participants completed a mock wing assembly task, and measures like completion time, error count, Net Promoter Score, and qualitative feedback were recorded. The HoloLens condition yielded faster completion times than all other conditions. HoloLens users also had lower error rates than those who used the non-AR conditions. Despite the performance benefits of the HoloLens AR instructions, users of this condition reported lower net promoter scores than users of the Tablet AR instructions. The qualitative data showed that some users thought the HoloLens device was uncomfortable and that the tracking was not always exact. Although the user feedback favored the Tablet AR condition, the HoloLens condition resulted in significantly faster assembly times. As a result, it is recommended to use the HoloLens for complex guided assembly instructions with minor changes, such as allowing the user to toggle the AR instructions on and off at will. The results of this paper can help manufacturing stakeholders better understand the benefits of different AR technology for manual assembly tasks.</p

Adaptive XR training systems design, implementation, and evaluation

Extended reality (XR) is a continuum encompassing various current and future virtual technologies such as augmented reality (AR) and virtual reality (VR). These technologies have been shown to increase learning outcomes when applied to training. For example, virtual and augmented reality have been shown to improve performance during assembly task training over traditional methods. However, even an advanced technology like XR won’t provide a benefit if the information is ill-timed or poorly suited to the problem. One potential way to solve this issue is with adaptive automation. Adaptive automation provides the ability to off-load tasks from a human to a computer. It has been shown to increase productivity with the added benefit of reducing human-labor costs. Previous research has shown that applying automation at the wrong levels or during the wrong stage of human information processing can result in decreased performance. Furthermore, if there are inaccuracies or gaps in the automated system, resulting in brittle automation, a human can develop an overreliance on the automation leading to costly mistakes. One place where adaptive automation is already being applied to XR technology is in the field of intelligent tutoring systems (ITS). Designers of XR ITS are increasingly automating the role of a human instructor in an effort to decrease the instructor to learner ratio. While works touting the development of this type of adaptive XR training systems exist, few address the design and efficacy of adaptations that do not directly pertain to the learning materials and pedagogy such as troubleshooting software and hardware use. Without this information, it is impossible for today’s researchers and developers to know how to design and implement these types of adaptive XR training systems. Therefore, this dissertation will identify and evaluate effective triggers and adaptations for XR training environments so that they can be replicated for future applications. This research sought to solve this problem in two stages. In the first stage, human XR instructors were used as models for potential adaptive automation schema. Since automation is the computer execution of a task once performed by a human, it was natural to choose human instructors as models for this work. Therefore, 11 semi-structured interviews were conducted with XR training and simulation experts. The questions posed during the interviews were crafted to determine how these instructors identified and mitigated confusion in learners. The information from these interviews was then analyzed for themes and synthesized with existing adaptive automation models. During the analysis phase, identifying and mitigating confusion were found to be analogous to the functions of triggers and adaptations in the adaptive automation model. These interviews resulted in two sets of recommendations. First, when designing adaptive triggers in XR, verbal triggers should be prioritized, followed by physical triggers. A head-mounted display (HMD) can be used to monitor verbal and physical triggers using a factory standard microphone and inertial measurement unit. Second, verbal and physical channels should also be prioritized when designing adaptation methods. This can be done through recorded audio messages, textual interfaces, and by providing demonstrative animations and models in an XR environment. Finally, adaptations with increasing levels of specificity and intrusiveness allow learners to solve problems independently. The second phase of this research implemented these recommendations within an XR task. A simple triangle completion task was chosen, in which a learner must teleport to three different positions in a VE using two different techniques. Afterwards, the learner is tested on their ability to remember where they started by pointing at the origin and clicking using an XR controller. Triggers and adaptations were developed to assist in the correction of five different erroneous behaviors during the task. These included incorrect button presses, inaction, and attempting to teleport to the wrong location. Finally, the resultant unsupervised, remote, adaptive XR simulation was evaluated to determine the efficacy of the adaptations and compared to a version of the simulation with no adaptations and a human instructor present to provide feedback, when necessary. The results of this experiment validated that the adaptations were sufficient at providing instructions to participants during the remote unmoderated study because the participant success rate was equal to that of the lab-based study (89%). In addition, participants’ performance and completion times were not statistically different from those of the lab group. The adaptations triggered when expected and had the intended effect of helping learners correct their mistakes. Finally, participants gave feedback about the intrusiveness, helpfulness, and quantity of adaptive feedback they received. It was found that the quantity of feedback was adequate without being too intrusive, however, there were mixed reviews about the subjective helpfulness of the feedback. Ultimately, this research was successful at increasing the efficacy of adaptive XR systems, and reducing the time and cost associated with humans facilitating XR training

Adaptive XR training systems design, implementation, and evaluation

Extended reality (XR) is a continuum encompassing various current and future virtual technologies such as augmented reality (AR) and virtual reality (VR). These technologies have been shown to increase learning outcomes when applied to training. For example, virtual and augmented reality have been shown to improve performance during assembly task training over traditional methods. However, even an advanced technology like XR won’t provide a benefit if the information is ill-timed or poorly suited to the problem. One potential way to solve this issue is with adaptive automation. Adaptive automation provides the ability to off-load tasks from a human to a computer. It has been shown to increase productivity with the added benefit of reducing human-labor costs. Previous research has shown that applying automation at the wrong levels or during the wrong stage of human information processing can result in decreased performance. Furthermore, if there are inaccuracies or gaps in the automated system, resulting in brittle automation, a human can develop an overreliance on the automation leading to costly mistakes. One place where adaptive automation is already being applied to XR technology is in the field of intelligent tutoring systems (ITS). Designers of XR ITS are increasingly automating the role of a human instructor in an effort to decrease the instructor to learner ratio. While works touting the development of this type of adaptive XR training systems exist, few address the design and efficacy of adaptations that do not directly pertain to the learning materials and pedagogy such as troubleshooting software and hardware use. Without this information, it is impossible for today’s researchers and developers to know how to design and implement these types of adaptive XR training systems. Therefore, this dissertation will identify and evaluate effective triggers and adaptations for XR training environments so that they can be replicated for future applications. This research sought to solve this problem in two stages. In the first stage, human XR instructors were used as models for potential adaptive automation schema. Since automation is the computer execution of a task once performed by a human, it was natural to choose human instructors as models for this work. Therefore, 11 semi-structured interviews were conducted with XR training and simulation experts. The questions posed during the interviews were crafted to determine how these instructors identified and mitigated confusion in learners. The information from these interviews was then analyzed for themes and synthesized with existing adaptive automation models. During the analysis phase, identifying and mitigating confusion were found to be analogous to the functions of triggers and adaptations in the adaptive automation model. These interviews resulted in two sets of recommendations. First, when designing adaptive triggers in XR, verbal triggers should be prioritized, followed by physical triggers. A head-mounted display (HMD) can be used to monitor verbal and physical triggers using a factory standard microphone and inertial measurement unit. Second, verbal and physical channels should also be prioritized when designing adaptation methods. This can be done through recorded audio messages, textual interfaces, and by providing demonstrative animations and models in an XR environment. Finally, adaptations with increasing levels of specificity and intrusiveness allow learners to solve problems independently. The second phase of this research implemented these recommendations within an XR task. A simple triangle completion task was chosen, in which a learner must teleport to three different positions in a VE using two different techniques. Afterwards, the learner is tested on their ability to remember where they started by pointing at the origin and clicking using an XR controller. Triggers and adaptations were developed to assist in the correction of five different erroneous behaviors during the task. These included incorrect button presses, inaction, and attempting to teleport to the wrong location. Finally, the resultant unsupervised, remote, adaptive XR simulation was evaluated to determine the efficacy of the adaptations and compared to a version of the simulation with no adaptations and a human instructor present to provide feedback, when necessary. The results of this experiment validated that the adaptations were sufficient at providing instructions to participants during the remote unmoderated study because the participant success rate was equal to that of the lab-based study (89%). In addition, participants’ performance and completion times were not statistically different from those of the lab group. The adaptations triggered when expected and had the intended effect of helping learners correct their mistakes. Finally, participants gave feedback about the intrusiveness, helpfulness, and quantity of adaptive feedback they received. It was found that the quantity of feedback was adequate without being too intrusive, however, there were mixed reviews about the subjective helpfulness of the feedback. Ultimately, this research was successful at increasing the efficacy of adaptive XR systems, and reducing the time and cost associated with humans facilitating XR training

Adaptive XR training systems design, implementation, and evaluation

Extended reality (XR) is a continuum encompassing various current and future virtual technologies such as augmented reality (AR) and virtual reality (VR). These technologies have been shown to increase learning outcomes when applied to training. For example, virtual and augmented reality have been shown to improve performance during assembly task training over traditional methods. However, even an advanced technology like XR won’t provide a benefit if the information is ill-timed or poorly suited to the problem. One potential way to solve this issue is with adaptive automation. Adaptive automation provides the ability to off-load tasks from a human to a computer. It has been shown to increase productivity with the added benefit of reducing human-labor costs. Previous research has shown that applying automation at the wrong levels or during the wrong stage of human information processing can result in decreased performance. Furthermore, if there are inaccuracies or gaps in the automated system, resulting in brittle automation, a human can develop an overreliance on the automation leading to costly mistakes. One place where adaptive automation is already being applied to XR technology is in the field of intelligent tutoring systems (ITS). Designers of XR ITS are increasingly automating the role of a human instructor in an effort to decrease the instructor to learner ratio. While works touting the development of this type of adaptive XR training systems exist, few address the design and efficacy of adaptations that do not directly pertain to the learning materials and pedagogy such as troubleshooting software and hardware use. Without this information, it is impossible for today’s researchers and developers to know how to design and implement these types of adaptive XR training systems. Therefore, this dissertation will identify and evaluate effective triggers and adaptations for XR training environments so that they can be replicated for future applications. This research sought to solve this problem in two stages. In the first stage, human XR instructors were used as models for potential adaptive automation schema. Since automation is the computer execution of a task once performed by a human, it was natural to choose human instructors as models for this work. Therefore, 11 semi-structured interviews were conducted with XR training and simulation experts. The questions posed during the interviews were crafted to determine how these instructors identified and mitigated confusion in learners. The information from these interviews was then analyzed for themes and synthesized with existing adaptive automation models. During the analysis phase, identifying and mitigating confusion were found to be analogous to the functions of triggers and adaptations in the adaptive automation model. These interviews resulted in two sets of recommendations. First, when designing adaptive triggers in XR, verbal triggers should be prioritized, followed by physical triggers. A head-mounted display (HMD) can be used to monitor verbal and physical triggers using a factory standard microphone and inertial measurement unit. Second, verbal and physical channels should also be prioritized when designing adaptation methods. This can be done through recorded audio messages, textual interfaces, and by providing demonstrative animations and models in an XR environment. Finally, adaptations with increasing levels of specificity and intrusiveness allow learners to solve problems independently. The second phase of this research implemented these recommendations within an XR task. A simple triangle completion task was chosen, in which a learner must teleport to three different positions in a VE using two different techniques. Afterwards, the learner is tested on their ability to remember where they started by pointing at the origin and clicking using an XR controller. Triggers and adaptations were developed to assist in the correction of five different erroneous behaviors during the task. These included incorrect button presses, inaction, and attempting to teleport to the wrong location. Finally, the resultant unsupervised, remote, adaptive XR simulation was evaluated to determine the efficacy of the adaptations and compared to a version of the simulation with no adaptations and a human instructor present to provide feedback, when necessary. The results of this experiment validated that the adaptations were sufficient at providing instructions to participants during the remote unmoderated study because the participant success rate was equal to that of the lab-based study (89%). In addition, participants’ performance and completion times were not statistically different from those of the lab group. The adaptations triggered when expected and had the intended effect of helping learners correct their mistakes. Finally, participants gave feedback about the intrusiveness, helpfulness, and quantity of adaptive feedback they received. It was found that the quantity of feedback was adequate without being too intrusive, however, there were mixed reviews about the subjective helpfulness of the feedback. Ultimately, this research was successful at increasing the efficacy of adaptive XR systems, and reducing the time and cost associated with humans facilitating XR training

Teleporting through virtual environments: benefits of navigational feedback and practice

Virtual environments (VEs) can be infinitely large, but movement of the virtual reality (VR) user is constrained by the surrounding real environment. Teleporting has become a popular locomotion interface to allow complete exploration of the VE. To teleport, the user selects the intended position (and sometimes orientation) before being instantly transported to that location. However, locomotion interfaces such as teleporting can cause disorientation. This experiment explored whether practice and feedback when using the teleporting interface can reduce disorientation. Participants traveled along two path legs through a VE before attempting to point to the path origin. Travel was completed with one of two teleporting interfaces that differed in the availability of rotational self-motion cues. Participants in the feedback condition received feedback about their pointing accuracy. For both teleporting interfaces tested, feedback caused significant improvement in pointing performance, and practice alone caused only marginal improvement. These results suggest that disorientation in VR can be reduced through feedback-based training.This version of the article has been accepted for publication, after peer review (when applicable) and is subject to Springer Nature’s AM terms of use, but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at DOI: 10.1007/s10055-022-00737-0. Copyright 2022 The Author(s). Posted with permission